Skip to content

feat: unconditional control-plane self-registration with no-auth support HYPERSHELL-297 - #284

Closed
markturansky wants to merge 10 commits into
mainfrom
docs/HYPERSHELL-297-managed-cluster-pull-model-spec
Closed

markturansky wants to merge 10 commits into
mainfrom
docs/HYPERSHELL-297-managed-cluster-pull-model-spec

Conversation

@markturansky

@markturansky markturansky commented Sep 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Every control plane now self-registers at startup unconditionally, whether or not OIDC authentication is configured
  • When auth is enabled, the managed cluster record is keyed on the JWT OIDC subject; when auth is disabled (local development), it is keyed on the cluster name alone with an empty oidc_subject
  • Adds managed-cluster-registrar Keycloak realm role and assigns it to the hypershell-control-plane service account so the JWT carries the required role claim
  • Includes platform spec updates that define the unified registration model

API server changes

  • Registration handler tolerates missing token (no-auth path)
  • New FindByNameNoOIDCSubject DAO method separates authenticated and unauthenticated identity spaces
  • Service uses dual-key advisory lock (oidc_subject or name: prefix) and conditional lookup

Control plane changes

  • Registration is unconditional at startup (no longer gated on OIDC + name both being set)
  • ManagedClusterName defaults to "local" when HYPERSHELL_MANAGED_CLUSTER_NAME is unset
  • Nil TokenSource omits Authorization header (safe interface-nil pattern)
  • 3 new unit tests for registration client (no-token, with-token, 403-forbidden)

Keycloak realm config

  • Added managed-cluster-registrar realm role
  • Added roles client scope to hypershell-control-plane so realm_access.roles appears in JWTs
  • Added service account user with the registrar role mapped

Test plan

  • go vet ./... and go test ./... pass on both api-server and control-plane
  • No-auth local dev: controller registers with name alone, empty oidc_subject, 201/200 idempotent
  • Auth-enabled Kind cluster: controller registers with OIDC token, oidc_subject populated from JWT sub claim, managed-cluster-registrar role enforced
  • ROKS hysh-ibm-01 validation (auth-enabled, route ingress mode)

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Advanced

Run ID: 54f84f12-2bd9-4d20-9959-e19f82ec5c4b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@amber-review-bot

amber-review-bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Collaborator

Amber review: changes requested

Amber review

Status: Complete

View the submitted review.

amber-review-bot

This comment was marked as outdated.

@amber-review-bot amber-review-bot added the amber/changes-requested Amber requested changes on this PR label Sep 15, 2026
amber-review-bot

This comment was marked as outdated.

@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

HyperShell environment destroyed

This ephemeral OpenShift environment has been destroyed. Comment /pr-extend to redeploy it.

amber-review-bot

This comment was marked as outdated.

@markturansky
markturansky marked this pull request as draft September 15, 2026 20:52
@markturansky markturansky changed the title docs(specs): add managed cluster pull model to architecture specs feat: unconditional control-plane self-registration with no-auth support HYPERSHELL-297 Sep 15, 2026
@markturansky
markturansky force-pushed the docs/HYPERSHELL-297-managed-cluster-pull-model-spec branch from e5da933 to 649cb59 Compare September 15, 2026 21:33
value: "hypershell-api-server.hypershell-system.svc.cluster.local:9000"
- name: HYPERSHELL_API_SERVER_URL
value: "http://hypershell-api-server.hypershell-system.svc.cluster.local:8000"
- name: HYPERSHELL_MANAGED_CLUSTER_NAME

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this HYPERSHELL_MANAGED_CLUSTER_NAME need to be defined in other deploys like openshift?

@markturansky
markturansky force-pushed the docs/HYPERSHELL-297-managed-cluster-pull-model-spec branch 2 times, most recently from 30e84bb to 13328c1 Compare September 24, 2026 14:46
@markturansky
markturansky marked this pull request as ready for review September 24, 2026 14:55
amber-review-bot

This comment was marked as outdated.

@markturansky
markturansky force-pushed the docs/HYPERSHELL-297-managed-cluster-pull-model-spec branch from 13328c1 to 8c78f2d Compare September 24, 2026 18:42
amber-review-bot

This comment was marked as outdated.

amber-review-bot

This comment was marked as outdated.

amber-review-bot

This comment was marked as outdated.

user and others added 9 commits September 25, 2026 11:05
…ERSHELL-297

Document the spoke-pull reconciliation mode in global-architecture,
control-plane, and data-model specs. The spoke control-plane runs on a
ManagedCluster, watches the Cloud Hub API server over gRPC, self-registers
via the idempotent registration endpoint, and reconciles only its own
gateways. Covers the OIDC chain (including the federation gap where the
ManagedCluster Keycloak is not yet federated), spoke gitops structure,
naming convention, gRPC external access (HYPERSHELL-333), and RBAC
uniqueness requirements.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…PERSHELL-297

Replace the hub-push vs spoke-pull two-mode framing with a single
unified model: every control plane self-registers via POST
/managed_clusters/registration at startup. A Cloud Hub's own control
plane is just another ManagedCluster. This works identically from local
development (fresh database, control plane registers locally) to
multi-cloud production deployments.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ocal-dev

Make registration unconditional in the specs: every control plane
self-registers with an API server at startup, identically in local
development and multi-cloud production. Identity is the OIDC subject when
authentication is enabled, else the name alone when the API server runs
with authentication disabled.

- managed-cluster-registration.spec.md: unconditional registration;
  JWT+role required only when auth is enabled; oidc_subject empty and
  upsert keyed on name in no-auth mode; renamed startup section to
  "Control Plane Startup and Loop"; added local-dev scenario; removed
  remaining spoke/hub control-plane terminology.
- control-plane.spec.md: unconditional startup path; corrected the
  cluster_id watch filter to the server-side behavior.
- data-model.spec.md: registration requirement covers the auth-disabled
  path; added local-dev scenario; qualified the 403 scenario.
- global-architecture.spec.md: Self-Registration and OIDC Authentication
  note the no-token local-dev path; tightened local-dev scenario.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ort HYPERSHELL-297

Every control plane now self-registers at startup regardless of whether
OIDC authentication is configured. When auth is enabled the record is
keyed on the JWT OIDC subject; when auth is disabled (local development)
the record is keyed on the cluster name alone with an empty subject.

API server: registration handler tolerates missing token, dao adds
FindByNameNoOIDCSubject, service uses dual-key advisory lock and lookup.
Control plane: registration is unconditional, nil TokenSource omits the
Authorization header, ManagedClusterName defaults to "local".
Keycloak: adds managed-cluster-registrar realm role, roles scope on
control-plane client, and service account role mapping.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…auth/scoping

- Update .forbidden-terms-whitelist.json line numbers shifted by prior edits
- Align registration idempotency scenario to (oidc_subject, name) key
- Rewrite cluster_id filtering as server-side (API server scopes streams
  to caller's identity, not client-side ignore)
- Add OIDC authentication requirement for externally exposed gRPC watch
  (HYPERSHELL-333 cannot hold the in-cluster JWT bypass when internet-facing)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…verlay

The controller self-registers on startup using this name. The Kind seed
and E2E tests look up the managed cluster by name "local-kind" to resolve
the cluster_id for gateway assignment. Without this, the controller
defaults to "local", creating a second managed cluster record whose
cluster_id does not match the seed's gateways.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…isioning contract

Replace the (oidc_subject, name) composite upsert key with name alone.
oidc_subject is now an audit field only - stored on first registration
when a JWT is present, but never used for record lookup.

This decouples cluster identity from the OIDC provider: any process with
the right name and managed-cluster-registrar role can register or recover
a control plane, multiple CPs per node are possible with distinct names,
and the responsibility for unique names moves to the provisioner (GitOps
or operator config) rather than being enforced via Keycloak subjects.

Changes:
- dao.go: remove FindByOIDCSubject/FindByNameNoOIDCSubject, add FindByName
- mock_dao.go: update mock to match interface
- service.go: single advisory lock on name, single lookup path, no 409 branch
- migration.go: new migration 2026092300000001 drops old composite index,
  creates uix_managed_clusters_name (name WHERE deleted_at IS NULL)
- plugin.go: register new migration
- specs: update upsert key wording throughout; replace 409 scenario with
  duplicate-name note; update design decisions table

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…t line

- Remove cfg.DatabaseProvider from startup log (field does not exist in Config)
- Wire newManagedCluster factory id param to Name so the name-unique index
  does not 409 on repeated test cluster creation
- Correct .forbidden-terms-whitelist.json vteam line from 1526 to 1525

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Six task bundles had their image content updated since the last
Renovate pass; the old digests were removed from the EC trusted-task
allowlist, causing the enterprise-contract check to fail.

Updated digests:
- task-clamav-scan:0.3.3
- task-init:0.4.3
- task-prefetch-dependencies:0.10.3
- task-push-dockerfile:0.3.1
- task-roxctl-scan:0.1
- task-rpms-signature-scan:0.2.2
- task-source-build:0.3.1

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…penShift overlay

The OpenShift seed script creates a ManagedCluster named local-openshift,
and the E2E test uses that cluster_id to create gateways. Without this
env var the control plane defaults to name=local, registers under a
different cluster_id, and never reconciles the E2E gateway - leaving it
stuck at phase=unknown.

Mirrors the Kind overlay which already sets local-kind.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

@amber-review-bot amber-review-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict

The earlier spec/security contradictions (client-side vs server-side watch filtering, external-gRPC auth) and the em-dash CI gate stay resolved, which is real progress. But the registration model still keys records on name alone while the specs promise identity-scoped watch authorization, and the unique-name migration still tightens a constraint with no dedup/backfill for pre-existing data, so the verdict stays REQUEST_CHANGES.

Summary

At this head the code path is clean (no panic, errors wrapped, no secrets logged, name validated as a DNS label, make check / forbidden-terms passes locally). The two Major concerns from the prior review are unchanged in the code, so this is still REQUEST_CHANGES; all findings already have Amber inline threads, which I link rather than duplicate.

Findings

[Major] Watch spec promises identity-scoped authorization the name-only registration model cannot provide. global-architecture.spec.md:548-551 says the API server scopes watch streams to the caller's cluster_id and "authorizes it against the caller's authenticated identity," and control-plane.spec.md:214 says the filter is server-side. But service.go:172-179 looks up the record by name and, on the existing-record branch, only updates LastSeenAt and returns the stored cluster_id; it never compares the caller's oidc_subject to existing.OIDCSubject, and oidc_subject is documented as audit-only (managed-cluster-registration.spec.md "Data Model Impact" / "Idempotent Registration"). With no durable identity->cluster_id binding, the server has nothing to authorize the watch's cluster_id against, and any managed-cluster-registrar holder can register another control plane's name, receive that cluster_id, and open its watch. This is a maintainer design decision: either bind the record to identity, or drop the identity-scoped-authorization claim. Existing thread: #284 (comment). Confidence: High on the inconsistency.

[Major] Unique-name migration tightens uniqueness with no dedup/backfill for pre-existing data. migration.go:47-51 (and the rewritten :21-25) creates CREATE UNIQUE INDEX ... uix_managed_clusters_name ON managed_clusters (name) WHERE deleted_at IS NULL. On main the only uniqueness was (oidc_subject, name), so two control planes with different subjects could already hold the same name. If such duplicate live rows exist in a database that ran the main version of this feature, the index creation errors, the migration aborts, and the API server fails to start. Add a dedup step before the index, or document why duplicates cannot exist in any running environment. Existing thread: #284 (comment). Confidence: Medium (depends on whether the main version ran against real data).

[Minor] An already-released migration is being edited in place. migrationAddRegistrationFields (ID 2026091000000001, migration.go:10-38) already shipped on main; rewriting its body to create the name-only index (:21-25) is dead code in any DB where it already ran (gormigrate skips by ID) - only the new migrationRegistrationKeyToNameOnly (ID 2026092300000001, :40-64) takes effect there - and it makes rollback inconsistent (the old migration's rollback now drops the name index while the new migration's rollback recreates the oidc_subject index). Revert this migration to its shipped definition and rely solely on the new migration. Existing thread: #284 (comment). Confidence: High.

[Minor] DefaultManagedClusterName = "local" is a silent collision footgun. config.go:31,103 defaults HYPERSHELL_MANAGED_CLUSTER_NAME to the literal local when unset. Combined with name-as-sole-key, any overlay that forgets to set this variable registers as local and shares one cluster_id with every other such control plane. The Kind (local-kind) and OpenShift (local-openshift) overlays set explicit names, but the code default is still local and there is no fail-fast, so ROKS / remote-managed-cluster overlays that omit the variable still collide silently. This also answers the open inline question on the Kind overlay (#284 (comment)): yes, every non-dev overlay must set a unique name; OpenShift now does, but the shared code default remains a trap. Consider failing fast when unset in an auth-enabled deployment. Existing thread: #284 (comment). Confidence: High.

Note (not a blocker): factory_test.go changing the test cluster Name from the shared literal "test-name" to the unique id is the expected fallout of the new global name-uniqueness constraint; it is fine, but it confirms the shared precondition was tightened.

Cross-PR coordination

A competing full implementation of this same feature is open in #362 ("mandatory control-plane cluster identity, hub gRPC TLS, cluster-scoped watch streams"). It edits the same functions and files this PR does (managedClusters/service.go, dao.go, migration.go, mock_dao.go, control-plane/internal/config/config.go, registration/client.go, and the same platform specs) but chooses the opposite registration/identity model. This PR renames the lookup to FindByName, makes name the sole upsert key, defaults it to local, treats a name collision as idempotent adoption of the existing record, and supports a no-auth path with an empty oidc_subject. #362 keeps oidc_subject as the owning identity, makes name+OIDC mandatory (removing HYPERSHELL_CLUSTER_ID), returns 409 on a name held by a different/empty subject, forbids silent adoption, and adds identity-based lookups precisely to authorize watch streams. These are mutually exclusive contracts for Register and for the DAO interface, and #362 is the concrete resolution of the identity-binding gap this review flags as a Major finding. Maintainers must decide which registration/identity model is canonical and in what order the two land; they cannot both merge as written.

A second decision is shared with #182 ("enforce management API JWT audience"). This PR's HYPERSHELL-333 section (global-architecture.spec.md:481-484) requires the externally exposed gRPC watch to enforce the caller's OIDC token and scope streams to its cluster_id, and states the in-cluster JWT bypass "cannot hold once the endpoint is internet-facing." #182 hardens management-API JWT audience validation while keeping the opposite contract in oidc-integration.spec.md (gRPC watch methods as a trusted in-cluster path, with a gRPC auth-bypass list). Maintainers must decide whether watch methods bypass JWT and reconcile the two specs before either lands.

A third coordination point exists with #185 ("periodic world synchronization"). It defines a "complete inventory" / paginated API-inventory reconcile pass that lists resources (including Gateway) and drives convergence and orphan cleanup from it, without cluster_id scoping on the inventory list. This PR establishes that each control plane owns only its cluster_id's gateways and must never reconcile or tear down another cluster's gateways, scoping both WatchGateways and the seed listing server-side by cluster_id. The two must agree on whether the world-sync inventory list is itself scoped by cluster_id; an unscoped "complete inventory" in the pull model could converge or clean up gateways outside a control plane's ownership. The owners should decide the canonical inventory-scoping contract before either lands.

Previous concerns

  • [Critical] CI blocker: em dashes + stale whitelist (#284 (comment)) - addressed. scripts/check_forbidden_terms.py exits 0 at this head (reproduced locally).
  • [Major] Client-side vs server-side cluster_id filtering (#284 (comment), #284 (comment)) - addressed. control-plane.spec.md:214 and global-architecture.spec.md:543-552 now describe a single server-side filter; the client-side "ignore the event" language is gone. (The residual identity-binding gap is tracked as a Major finding above.)
  • [Major] External gRPC watch lacks authN/authZ (#284 (comment)) - addressed. global-architecture.spec.md:481-484 now requires the external gRPC watch to enforce the OIDC token and scope streams to the caller's cluster_id.
  • [Minor] Idempotency key stated name-only (#284 (comment)) - addressed as an intentional design change. managed-cluster-registration.spec.md documents name as the sole upsert key; spec and code are internally consistent. The security trade-off is raised as a Major finding above for a maintainer decision.
  • [Major] Name-only upsert has no identity check on takeover (#284 (comment)) - still present (service.go:172-179 updates only LastSeenAt and returns the stored cluster_id, no oidc_subject comparison).
  • [Major] Unique-name index has no dedup/backfill (#284 (comment)) - still present (migration.go:47-51).
  • [Minor] Editing an already-released migration (#284 (comment)) - still present (migration.go:10-38).
  • [Minor] Defaulting the registration name to local (#284 (comment)) - still present (config.go:31; the OpenShift overlay now sets local-openshift, but the code default and lack of fail-fast remain).

Findings Summary (ordered by severity, highest first)

  1. [Major] Watch spec promises identity-scoped authorization the name-only registration model cannot provide (takeover via name) - Security / Spec Consistency (global-architecture.spec.md:548-551; managed-cluster-registration.spec.md; service.go:172-179)
  2. [Major] Unique-name migration tightens uniqueness with no dedup/backfill for pre-existing data - Migration Safety (migration.go:21-25,47-51)
  3. [Minor] Already-released migration edited in place (immutability) - Convention (migration.go:10-38)
  4. [Minor] DefaultManagedClusterName="local" collision footgun for overlays that omit the variable - Config / Robustness (config.go:31,103)

Convention Checklist

Convention Result
No em dashes / forbidden terms (make check) Pass
No panic() in production code Pass
Errors wrapped with fmt.Errorf(...%w) Pass
No secrets in logs or responses Pass
Input validated (DNS label) Pass
Migration safe for pre-existing data Fail
Migration immutability respected Fail
Identity-scoped authorization consistent across specs Fail
Conventional commit messages Pass

@markturansky

Copy link
Copy Markdown
Collaborator Author

closing in favor of #362

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amber/changes-requested Amber requested changes on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants